Arksey and O′Malleyʼs consultation exercise in scoping reviews: A critical review

Abstract Aims To explore how consultation exercises were described in a convenience sample of recent scoping reviews. Design Critical literature review. Data sources We searched PsycINFO, Embase, CINAHL and PubMed in July 2020. Our inclusion criterion was a peer‐reviewed journal article reporting a scoping review in Danish, English, Norwegian or Swedish. Review methods We identified a convenience sample of articles (n = 66) reporting a consultation exercise as part of a scoping review. The descriptions of the consultation were charted, summarized and critically discussed. Results The current analysis showed no widely accepted consensus on how to approach and report a consultation exercise in the sample of scoping reviews. The reports of stakeholder consultation processes were often brief and general, and often there were no reports of the effects of the stakeholder consultation processes. Further, there was no discussion of the principal theoretical problems mixing stakeholder voices and review findings. Conclusion The finding that conventional research ethics and research methods often were suspended could indicate that the stakeholder consultants were in a precarious position because of power imbalances between researchers and stakeholder consultants. We suggest that a consultation exercise should only be included when it genuinely invites participation and reports on the effect of alternative voices. Impact Scoping reviews are common across a range of disciplines, but they often lack definitional and methodological clarity. In their influential approach to scoping studies, Arksey and OʼMalley introduced an optional ‘consultation exercise’, which has been heralded as a valuable tool that can be used to strengthen the process and outcome of a scoping study and to support the dissemination of the studyʼs findings and its implications. However, there is no clear outline on about how to operationalize consultations of stakeholders in scoping studies/reviews. This article includes recommendations for consultation exercises, including encouraging an aspirational move from ‘consultation’ to ‘participation’.


| INTRODUC TI ON
Scoping reviews are common across a range of disciplines and more recently, guidelines for conducting scoping reviews (Peters et al., 2020) and reporting scoping reviews (Tricco et al., 2018) have been published. However, scoping reviews often lack definitional and methodological clarity (Colquhoun et al., 2014;Khalil et al., 2016;OʼBrien et al., 2016). In their widely cited paper, Arksey and OʼMalley (2005) outlined an approach to scoping studies that included an optional consultation exercise that could 'enhance' the results. In this paper, we will focus on the use of stakeholder consultations in scoping studies/reviews, and following a critical review of current consultation practices, we will recommend on how to design and report stakeholder consultations in ways that can substantially strengthen scoping study/review processes and outcomes.

| BACKG ROU N D
For Arksey and OʼMalley (2005), a scoping study is a type of literature review that differs from systematic reviews by addressing relatively broad topics and not asking very specific research questions (p. 20).
In their paper, they focused on scoping studies whose purpose was to identify gaps in the existing research literature and research activity. Like systematic reviews, the scoping study/review process was described as rigorous and transparent procedures that others ultimately should be able to replicate, while at the same time iterative and non-linear. Arksey and O′Malleyʼs outline included five stages, from identifying the research question to collating, summarizing and reporting the results (pp. 22-28). The consultation exercise was an optional sixth stage that could 'inform and validate findings from the main scoping review' (p. 23).
With regards to the consultation exercise, Arksey and OʼMalley (2005) were influenced by Oliver (2001), who described opportunistic and heuristic 'consultations' (p. 175) as part of a systematic review process that challenged conventional research knowledge. As part of a workshop for health promotion practitioners, Oliver presented a systematic review of smoking cessation programs for pregnant women written by Lumley. The presentation of the review was not received well by the workshop participants who felt that too much had not been taken into account in the reviewʼs conclusions. Oliver and the workshop participants decided to send a letter describing their concerns to Lumley, who subsequently invited Oliver to participate in an update of the review. As part of updating the review, Oliver spoke to pregnant women about smoking through a 'rapid appraisal exercise ' (2001, p. 173), which created a different, more holistic awareness about the programs and their impacts on smokers and their babies as well as an awareness of the shortcomings of the dominating health research paradigm. Oliver (2001)  It is important to notice that Arksey and OʼMalley (2005) described the design of a scoping 'study', not simply the design of a scoping 'review'. Their study design was in effect a mixed-methods design involving a literature study (the scoping review) and an interview study (the consultation exercise), and they argued that the consultation exercise could create 'added value' to both the literature study and the findings of the overall scoping study (p. 29). However, it remained unclear when and how to consult with stakeholders and how to integrate information from the consultation exercise with the review findings in the overall study.
In a subsequent methods discussed, Levac et al. (2010) addressed some of these issues and recommended that consultation should be a required part of scoping studies and have a clear purpose that could include sharing preliminary findings with stakeholders, validation of findings and informing future research. They suggested preliminary review findings as a fruitful outset for consultation and recommended that authors provided clear descriptions of the type of stakeholders that would be consulted, how data would be collected during the consultation exercise, and how consultation exercise data would be integrated into the overall study. Levac et al. (2010) emphasized consultation as an opportunity for 'knowledge transfer' with stakeholders in the field that could potentially strengthen dissemination of the findings (p. 7). In the final part of their discussion, Levac et al. (2010) problematized different terminology around scoping approaches, which we believe highlighted a central design issue: Using the term 'scoping study', as originally suggested by Arksey and OʼMalley (2005), indicates a mixed-methods design where findings from a distinct literature review and findings from a distinct consultation exercise are integrated in an overall scoping study design, whereas 'scoping review' indicates a design, where data from a consultation exercise are merged into a scoping review process. Levac et al. (2010) suggested using Arksey and O′Malleyʼs (2005) terminology, which is more precise. However, this suggestion has not been taken up widely, and most 'scoping studies' are in fact referred to as 'scoping reviews'. In the current paper, we will refer to 'scoping

Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. recommendations for consultation exercises, including encouraging an aspirational move from 'consultation' to 'participation'.

K E Y W O R D S
community participation, health services research, methods, nursing, review studies' when referring to Arksey and OʼMalleyʼs methods and terminology, and to 'scoping review' or 'scoping study/review' when referring to the wider field of literature reviewing. Daudt et al. (2013) reflected on their use of Arksey and O′Malleyʼs (2005) framework and compared their own approach to the recommendations made by Levac et al. (2010). Most notably, Daudt et al. (2013) showed that the engagement of a large interprofessional team had been helpful for them, which in effect meant that they had had stakeholders involved throughout the whole process and that the previous distinction between research team and stakeholders was no longer clear. In line with Levac et al.'s (2010) recommendations, stakeholdersʼ contributions were described as useful for guiding future research as part of a knowledge translation process.
The stakeholder consultation in scoping studies/reviews has been examined as part of two 'scoping reviews of scoping reviews' (Pham et al., 2014;Tricco et al., 2016). Pham et al. (2014)  reported consulting a librarian as part of their search strategies and that less than 10% of the sample reported an 'end-of-grant' knowledge translation . We find these data and conclusions on the use of consultation exercises very general and therefore difficult to interpret. For instance, how should we consider studies that mention a stakeholder consultation without providing any evidence of the process and outcome?

| Aim
In the literature reviewed above, there is a general consensus around the consultation exercise being a valuable tool that can be used to strengthen the process and outcome of a scoping study/review and to support the dissemination of the study/reviewʼs findings and its implications (Arksey & OʼMalley, 2005;Levac et al., 2010;Peters et al., 2020). However, there is no consensus about how to bring data from consultations of stakeholders into scoping studies and reviews. The aim of the current paper was to explore how consultation exercises are described in a convenience sample of recent scoping reviews and to provide recommendations for future consultation exercises in scoping studies and reviews.

| Design
We were guided by three research questions: (1) At what stage of scoping studies were stakeholders consulted? (2) Who were the stakeholders and how were they consulted? and (3) How was information from stakeholders integrated into scoping studies?

| Search methods
We used a relatively simple search strategy to identify a convenience (non-probability) sample of articles reporting a consultation exercise as part of a scoping study/review. Our inclusion criterion was a peer-reviewed journal article reporting a scoping study/review and published in Danish, English, Norwegian or Swedish. We searched PsycINFO, Embase, CINAHL and PubMed, because of these four databasesʼ collective scope. There were no limitations on dates of publication. The literature searches took place in July 2020.

| Search outcome
The searches in PsycINFO, Embase and PubMed were identical and combined the free text searches, 'scoping review' AND ('consultation*' OR 'knowledge translat*') in 'all text'. In CINAHL, we added the controlled heading, 'scoping review' to the search string. These searches identified 68 references in PsycINFO, 396 references in Embase, 211 references in CINAHL and 369 references in PubMed.
We screened the studies using the following exclusion criteria: (1) Scoping studies that did not explicitly pay homage to the methods originally outlined by Arksey and OʼMalley (2005), (2) studies that did not include a mention of a consultation exercise, (3) mixedmethods papers where details of the literature review were not sufficiently reported or the consultation exercise was reported in a different publication and (4) study protocols. To support the transparency and reliability of the study selection, we used the Covid ence.org software to collaborate and document the search process.

| Data abstraction
First, each of the included articles was read by two reviewers who independently extracted and discussed data with regards to, (1) Where in the process stakeholders participated in the scoping review?, and (2) How their participation influenced the publication.
Differences were discussed until a consensus was reached, which in some situations involved a third reviewer. Second, guided by our research questions, groups of two to three reviewers each focused on one of the three research questions by extracting and summarizing specific information from all the included papers. The groups made use of an additional reviewer, if they could not reach a consensus on how to read and interpret the description of the stakeholder consultations. This was needed because papers often lacked details on how consultation took place and what it resulted in. Third, data were tabulated and discussed by all authors.

| RE SULTS
Reports of stakeholder consultation processes were often brief and general, and often there were no reports of the effects of the stakeholder consultation processes.
Consultation took place in different stages of the review processes (see Table 1), with 'consultation during literature searches' (n = 24) and 'consultation about interpreting findings' (n = 45) as the most frequently reported, and 'consultation during preparation phase' (n = 11) and 'consultation after completion of review' (n = 11) as the less frequently reported. A small number of studies reported consultation throughout the whole review process (n = 8).
Moreover, in the preparation stage, the most frequently reported use was 'to generate research questions'; in the literature searching stage it was 'to identify relevant studies'; in the findings stage it was 'to get feedback/discussion of findings', and in the after full review stage they were 'to ensure user (practice or knowledge) relevance', 'to get feedback on review findings' and 'to inform further research'.
An effect of the consultation exercise was reported in 80.3% (53/66) of the studies. We categorized any description of an outcome related to the consultation exercise-no matter how modest-as an effect. Although this threshold resulted in a relatively high rate of observations of effect, we believed this was a more reliable approach than trying to define effect as more than a mere mention.
The descriptions of consultations tend to be rather general. The number of stakeholder consultants varied from 1 to >300 with a median number of 9. In 38% (25/66) of the papers, the number of stakeholder consultants was not provided. There was considerable variety in the methods used in the consultation exercises, related to whether the consultation was conducted individually, 21% (14/66), for instance interviews, and surveys, or in groups, 26% (17/66), for instance focus groups, workshops, public discussion, expert panels and advisory groups. Further, 14% (9/66) was mixed between individual and group-based methods. In the remaining 39% (26/66) information about methods was not provided.
There was variation in terms of having a consultation once, 41%

| DISCUSS ION
The current analysis showed no widely accepted consensus on how to approach and report a consultation exercise in the sample of scoping reviews. Furthermore, there was significant variation in the design of consultation exercises and stakeholders selected for the consultation exercises. There was no general agreement on who was qualified to be a stakeholder consultant.
Heavily influenced by Oliver (2001), Arksey and O′Malleyʼs (2005) optional consultation exercise introduced user consultation as part of a literature review method, which could be interpreted as a response to changing healthcare policies with increased pressure towards democratizing healthcare knowledge and practice through public participation (National Health and Medical Research Council, 2016).
However, user participation is often criticized for being tokenistic with aspirational policies ahead of changing practices (Sangill, Buus, Hybholt, & Berring, 2019), and the current review showed that the practice and effects of consultation exercises were rarely reported in meaningful detail. While we cannot rule this out as a result of lowprioritized or poor reporting, we believe this is indicative of a general tokenistic approach to user participation.
Also following Oliver (2001)  The finding that the scoping reviews rarely reported their consultation exercises in meaningful detail could indicate that the stakeholder consultants are in a precarious position because of power imbalances between researchers and the stakeholder consultants. First, consultation exercises were most often reported without sufficient reference to the research methodology. As a consultation exercise adds legitimacy to the findings of a scoping review, the trustworthiness of the contribution of the consultation should be transparent, rigorous and possible to assess by readers.
This would, for instance, include methods to ensure balanced results through recruitment processes and analysis/interpretive processes.
Stakeholders can ultimately be presented as 'agreeing' proponents of the results of a scoping review by researchers who have not rigorously researched and validly represented their situated perspectives.
Second, some studies considered the consultation as a formal data collection and reported obtaining approval from a research ethics committee to conduct the consultation exercise, and other studies reported gaining informed consent from stakeholder consultants. Other studies did not regard the consultation as a formal data collection and stated that ethical approval and informed with stakeholder consultants (Berring, Buus, & Hybholt, 2021).
However, since consultation exercises can be characterized as a type of participatory research, reflections on ethical issues are highly important in that stakeholder consultants might be potentially vulnerable groups, for example patients or service users.
When stakeholder consultants become co-researchers, researchers are accountable for ensuring the protection and autonomy of stakeholder consultants (Tee & Lathlean, 2004).
Arksey and OʼMalley (2005)  where results from one discreet study (a scoping literature review) is followed up by a second discreet study (data collection in a consultation exercise) (Creswell & Plano Clark, 2007). However, our analysis also identified integrated designs where stakeholder consultantsʼ comments were juxtaposed or mixed with review findings. Such merging of expertise, data and knowledge fundamentally challenges modernist views on methods-based rationality and scientific knowledge (Haraway, 1988). We often showed that the knowledge produced in the reviewed papers fruitful, but we are concerned about a general lack of researcher reflexivity in research processes, and a lack of theorizing the fuller consequences of the methodological and ethical implications of consultation exercises.
We interpret Arksey and O′Malleyʼs (2005)  As the analyses were based on a convenience sample of reported scoping reviews, we make no claims about the exact proportions of observations relative to 'scoping reviews' in general. The size and character of the sample allowed us to identify patterns, and our meticulous joint search and study selection practices were designed so that they would not add to skewing the sampleʼs characteristics.
Further, as our general findings had substantial similarities with previous research (Pham et al., 2014;Tricco et al., 2016), we believe that our findings are relevant beyond the concrete analysis.
Mapping the reported consultation exercises was unexpectedly challenging because of limited and complex descriptions and we relied on several iterations of collaborative interpretations to be confident in the process. To provide transparency into the interpretations, we provide our raw mapping in the supplementary material.

| CON CLUS ION
Despite the fact that all of the articles in our sample were structured using Arksey and O′Malleyʼs (2005) six stages, the reported consultation exercises generally lacked descriptions of specific aims, designs and effects, which left the impression that simply stating that a consultation exercise had been done was a quality mark in itself rather than considering its potential value add.
Scoping reviewsʼ consultancy exercises need a re-invigorating update, as they were largely tokenistic user participation practices.
Considering the transformation of practices over time, we notice that Oliverʼs (2001)  Influenced by Arnsteinʼs (1969) ladder of citizen participation, we argue that scoping reviews should climb the rungs of the ladder and include full participation by alternative stakeholders, rather than 'just' consulting them. However, for Arnstein, citizen participation was narrowly focused on power and control in decision-making (Tritter & McCallum, 2006), and we believe that user 'participation' in literature reviewing can include other dimensions than a transfer of power, such as respectfully sharing different experiences and appropriately engaging with alternative, historically excluded, voices (as highlighted by Oliver).
Here, we close by recommending some principles for the development of reporting consultation exercises, which can provide transparency and align expectations to the role and contribution of the stakeholder consultants: • Provide arguments as to how the selected stakeholders for the consultation exercise bring important different perspectives into the review process.
• Provide a comprehensive account of both the process and outcomes of the consultation exercise, including critical reflections on how the stakeholder perspectives from the consultation exercise challenged conventional perspectives.
• Treat the consultation exercise as research by: • following and reporting widely accepted approaches to research design and methodology, including data collection and data analysis.
• following ethical principles for the protection of stakeholdersʼ interests and rights as research participants in research, such as giving informed consent.
• Treat the consultation exercise as part of a mixed-methods study design, discuss the epistemological consequences of working with mixed methods, and report accordingly.

ACK N OWLED G EM ENTS
Open access publishing facilitated by University of New South Wales, as part of the Wiley -University of New South Wales.
[Correction added on 21 May 2022, after first online publication: CRUI funding statement has been added.].

CO N FLI C T O F I NTE R E S T
The authors report no conflicts of interest.

PEER R E V I E W
The peer review history for this article is available at https://publo ns.com/publo n/10.1111/jan.15265.

DATA AVA I L A B I L I T Y S TAT E M E N T
Our data can be made freely available as supplementary material.